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This report on guidelines for research reporting was, presented by the 
senior author at a symposium on "Problems with Meta-Analysis of Decision- 
Making, Research" at the 13th Annual Meeting of the American Institute for 
Decision Sciences on November 18, 1981, in Boston, Massachusetts. The guide- . 
lines were developed by the authors as a consequence of their experiences 
in conducting a meta-analysis of the career counseling outcome research 
published Wuring 1950-1980. A substantial .part of that published research 
overlaps with the research reported in ARI Technical Paper 316. This 1978° 
report, "Outcome Measures for Career Counseling Research," was authored by 
Laurel mW. oie under Army Project 2Q762717A766. 
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SUFFICTENCY IN THE REPORTING OF RESEARCH RESULTS: SOME GUIDELINES 


Requirement: 

The US Army Research Institute for the Behavioral and Social 
Sciences (ARI) conducts research on issues of interest to the Army. 
Similar research may be accomplished by other services, Government _ 
agencies, and in the civilian community. Over time, a substantial 
amount of research in the same area may accumulate. “The problem then 
becomes one of integrating those research results in a way shat is 
meaningful for Army decision makers. The meta-analysis approach used 
by Glass (1976) and his colleagues has proved useful for integrating 
the results of many studies. 


Procedure: ; 
¢ 


The authors of this report are conducting a meta-analysis of the 
career counseling outcome research published during the period 1950- 
1980. They have found that the data:‘needed for this type of research 
integration are not always available in research reports or published 
articles. Accordingly, the authors have developed some guidelines for 
research reporting which, if followed, would facilitate research inte- 
gration. 


Findings: : 


Inadequacies of research reporting encountered by research inte- 
grators include: failure to provide needed data, selective reporting 
of data, and incomplete descriptions of samplesAor interventions. 
Attempts to obtain needed information from authors are generally un- 
successful, especially if, considerable time has elapsed since the re- 
search was conducted. The guidelines the authors present are: (f) 
Report means and standard deviations (or correlation coefficients), 
and N's for all groups; (2) Report exact levels of significance (if 
available) and degrees of freedom for significance tests; (3) Organize 
data for analysis by outcome variable and do not conduct separate analy- 
ses for evéry item in an instrument; (4) Report nonsignificant as well 
as significant findings and report results for all groups included in 
the analysis; (5) Insure accuracy of data. For the purpose of using 
‘ research results for organizational decision making, it is the respon- 
sibility of the research integrator to portray the results of the meta- 
analysis in a realistic manner. 


vii aS 


Utilization of Findings: 
The purpose of this report is to encourage researchers to report 
‘ their’ research results in a manner that will make their results more 
useful in a meta-analysis. Given adequate reporting of .research results ? 
and meticulous implementation of the meta-analysis approach, decision makers 


will have access to more valid information on which to base their decisions. 
~ 
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SUFFICIENCY IN THE REPORTING OF RESEARCH RESULTS: SOME GUIDELINES 


INTRODUCTION 


At a meeting of the American Psychological Association (APA) a little 
over a year ago, Frank Schmidt stated that “the most important problem in 
psychology and the social sciences today is the failure to produce cumulative 
knowledge” (Schmidt, 1980). We agree with Dr. Schmidt. For decades, we 
have conducted research on a wide variety of organizational topics. These 
have included personnel selection, leadership and management, socio-technical 
systems, and many others. Some of these topical areas are directly relevant 
to the decisions organizations must make in allocating and developing their 
resources. In many instances, howéver, we have discovered that this 
plethora of research leaves us in a confused state. For example, many 
authors find it difficult to arrive at general summary statements which - 
capture the essence of the results and make these results useful to the 
reader. Thus, as researchers we seem to have produced a great many findings, 
but we have usually not integrated the results of our research in a meaningful 
way. 


Reviewers who use traditional narrative or literary approaches to 
integrating the results of many studies have based their conclusions on 
their personal reading of a set of studies on a given topic. The conclusions 
drawn from these customary reviews are necessarily subjective and may be even 
more so if the reviewer has excluded certain studies deemed methodologically 
inferior or otherwise inappropriate. Some reviewers employ the more 
sophisticated “vote-counting"” (or “box-score") approach in which the 
results of each study are sorted into positive significant, negative 
significant, and nonsignificant categories and bases his or her conclusions 
on the resulting tallies. Although more systematic, and certainly more 
revealing when dealing with a considerable number of studies, the vote- 
counting approach does not take into account the size of the effects 
tabulated and, as Hedges and Olkin (1980) have shown, may still result in 
biased estimates of outcomes. 


Traditional methods are not adequate for reviewing the results of 
a sizable body of resarch. In integrating the findings of career counseling 
outcome research, we are following the lead of Glass (1976) and others by 
employing techniques of “meta-analysis.” Meta-analysis provides a means of 
quantifying and combining results of individual studies. The unit of 
analysis is a standardized mean difference known as the “effect size.” As 
Smith and Glass (1977) and Smith, Glass, and Miller (1980) have defined it 
in applying meta-analysis to the psychotherapy research, the effect size is 
the difference ‘between the means of the experimental and‘the control groups 
divided by the standard eeviation of the control groups on a dependent 
variable. That is, } 


Mem Me 


SD¢ 


Thus only three statistics are needed to calculate an effect s&#zeé: mean 
of the experimental group, mean of the control group, and standard deviation 
of the control group. We naively assumed that these basic data fiust surely 
be reported in every study. Not so! As others have ‘found before us, we 
discovered a surprising number of studies that contained data which were 
insufficient for the application of the usual méta-analysis techniques, ! x 
Our efforts to obtain missing data from the authors: reporting the research 
met with only. partial success. The longer the lapse of time since the study s 
was published, the more difficult it was to secure the data we needed. 
2 4 ‘ 

Although the content of our meta-analysis is of ost interest to 
those in the field of career counseling, the problem this paper addresses 
is the inadequacy of research reporting. Our experiences have convinced us_ 
that there should be certain basic requirements for reporting research. a) 
With the proliferation of research in almost everyarea’of investigation, 
we’ believe the integration of research results will become an -increasingly 
important task. Thus, it is important for authors to report their research 
‘results in a manner that will facilitate efforts to combine the results of 
many studies. The\purpose of this paper is to.offer- a, i guidelines 
which we believe will make their, research findings direct usable in any 
subsequent meta-analysis. ‘ : 


GUIDELINES FOR RESEARCHERS REPORTING RESULTS 


Commonly Encountered Problems . - 


A number of problems may ensue when one embarks on a project of 
retrieving data from research reports in order to aggregate the results. 
In our réview of the career counseling outcome research published during 
the period 1950-80 (Oliver & Spokane, 1981), we encountered a number of 
difficulties. These problems included the failure to provide needed 
data,2 selective reporting of data, errors, incomplete descriptions of 
“interventions or samples, and the difficulty of obtaining required data 


IThere are ways in which effect sizes can be estimated when final status 
and standard deviations are not reported. -However, these are estimates P 
¢ 


calculated from other data which may or may not be reported. See Glass, 
McGaw, \and Smith (1981) and Rosenthal (1980) for details. 


2Some o&{ the articles we included in our integrative analysis of the 
career co ling outcome literature came from journals whose audience 
consists primarily of counselor practitioners. Such articles sometimes 
report results as “statistically significant” and do not include means, 
standard deviations, or the level of significance. Our position is that 
authers of this type of article could, in most cases, include a small 
supplementary table or, at the least, make their basic data available to 
other researchers. 


,“from authors., The statistic most frequently omitted was the standard 
deviation. Occasionally, authors failed to report N's. We also found a 
number of obvious errors in table (We do not, of course,. know how many 
unobvious errors we ,failed to détect.) Sometimes the descr{ptions of the 
career interventions were so limited that it was difficult to determine, 

what kind of an intervention was intended (let alone replicate such an 
intervention from the description). Inadequate descriptions were particularly 
troublesome in categorizing control conditions. We had decided to classify 
control groups as “controls” only if they had received no career intervention. 
_ Some control groups (especially those labeled “placebo” controls).had in 

fact received a minimal intervention. And it was sometimes not possible -to 
tell from the author's description how the subjects in such groups had been 
Handled. Where feasible, we requested (by telephone or*mail) the required 
information. Such efforts were generally not successful. People move, 

die, throw out data, or cannot find data they believe they have kept. We 

have concluded that there is no adequate substitute for a complete presentation 
in the research report. 


Suggestions for Authors (and Editors) 


In this section, we present some suggestions we believe would render 
research reports more usable for research integration. By research reports, 
we mean journal articles, organizational reports of research, and conference 
papers. 


1. Basic data. At the very least, researchers should report means 
and standard deviations (or correlation coefficients if a correlational 
analysis) plus N's for all groups. While it would also be helpful to have 
these data given for the entire sample, we realize that journal space 
limitation may preclude this course of action. However, these data could 
easily be Included in the original institutional report of the research (if 
there is one). In any case, they should be obtainable from the author of 
the article. If change scores are used,” the pretest and posttest data 
(means and standard deviations) should be reported as well. If analysis of 
covariance is used, the correlation between the covariates and the dependent. 
variables should be reported. It is also desirable for authors to report 
unadjusted posttest means and. standard deviations if no pretest differences 
among groups were found. Editors and reviewers who wish to conserve costly 
journal space should be careful not to displace these basic data. 


2. Significance tests. In cases where significance tests have been 
made, authors should report F ratios or t values (or correlation coefficients) 
with their exact probability levels (not p> -05," but p = .13"), if available, 


“ 
3Becduse of the problems associated with change (difference/gain) scores, 
we would not encourage researchers to use them. Kngpp (1980) has discussed 
the unreliability of change scores in counseling rfsearch. See the Cronbach 
and Furby (1970) review article and Harris' (1963) book for more detailed 
discussions of measuring “change.” 
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and degrees.of freedom. It was our experience that when no significant 
differences resulted, authors sometimes neglected to report those non- 
significant results and merely settled for a statement of "no significant 
difference was found." 


3. Organization of data for analysis. Another dilemma facing the 
would-be research integrator is the way authors organize and analyze their 
data. We encountered a number of studies in which an analysis was conducted 
on each item in a questionnaire with data reported only for individual 
items. .In such cases, we feel that items should be grouped by type of 
dependent variable and data also reported for each cluster of items. Al 
items relating to career information seeking, for example, should be 
subsumed under that variable. Even if subscales of a test or inventory are 
of individual interest, a total score should generally be reported for the 
instrument, where appropriate. as 


4. Selective reporting of data. In a few instances, authors chose 
to report only part of the data generated from a study. Althaugh this 


practice is certainly acceptable when a study is too large to report in its 
entirety, failure to report nonsignificant findings or deletion of results 
for certain groups makes proper research aggregation impossible. We believe 
authors should report results for all groups on all dependent variables. 


5. Making data available. As we noted earlier, published articles 
may be directed to a practitioner audience rather than to researchers. 
Such articles often do not contain the basic data needed to calculate 
effect sizes. If it is clearly inappropriate or space limitations do not 
permit the inglusion offa table reporting means, standard deviations, and 
N's for all groups, we encourage authors to m&ke these basic data available 
to researchers in some alternate fashion. The difficulty of the “make _ 
available” procedure is that it is not always possible to contact authors, 
especially if a considerable period of time has elapsed since the publication 
of the article. Accordingly, we urge editors to permit the inclusion of 
such a summary table, perhaps as a short appendix. For most research 
published in practitioner ournals, these tables should not be very extensive. 
Oliver and Spokane (1981) found that 70 percent of the career counseling 
outcome studies they surveyed employed only one or two dependent variables. - 


6. Accuracy of data. From time to time, we noticed errors in articfes. 
We suggest authors completely recheck data in the final versions of their 
reports against the computer printouts containing their results. If an 
article is revised, the final revision should again be checked against the 
printout. Page proofs can then be checked against the final version. (We 
of course believe authors should check and recheck their entire papers, 
but here we are focusing on the data required for meta-analysis). 


BEYOND SUFFICIENT REPORTING OF RESULTS 


We discovered a number of shortcomings in the published research in 
career counseling as we attempted to integrate the results across a large 
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number of studies. We have reported these deficiencies in the hope that 
future reviewers might encounter fewer stumbling blocks. We realize 

that our suggestions, even if heeded, will not be reflécted in the content 
of journal articles for a considerable period of time. An even longer 
period would be required -for the findings to be incorporated into an 
integrative analysis. Garvey (1979) has suggested that the process of 
disseminating and communicating findings, from the time a given study is 
completed until it appears as a part ofa body of knowledge in a textbook 
takes 13 years. We made use of the data available to us, however limited, 
’in our analysis of the effects of career counseling. : 5 


We do want to comment briefly in two areas, namely, the applications 
of such aggregate findjngs in the decision-making process and, secondly, 
the limitations of meta-analysis that argue for restricting generalizations 
about the findings. 


Making Use of ‘Integrative Findings 


Glass and Smith (1980) have described meta-analysis as "nothing more 
than the attitude of data analysis applied to quantitative summaries 
of individual experiments” (Glass and Smith, 1980, p. 277). In our 
experience, however, the set of statistical techniques embodied in the 
meta-analysis approach are employed to derive summary statements with 
respect to a body of'research. Because these techniques are increasing in 
popularity and, in light of some cogent criticisms of meta-analytic procedures 
that have recently been proposed, we feel that it is important to delineate 
areas in which these findings can be usefully employed. Here, we rely 
substantially on points made by Pillemer and Light (1980). ‘ 


Translating the results of empirical research to non-scientists. 
A number of groups may have interest in the conclusions drawn from inter- 
grative reviews. In the field of career ee this might include a 
large and diverse group of professional practitioners of varying levels of 
sophistication. These consumers may wish to extract a germ from the 
evidence without having to wade through a profusion of individual studies. 
Other groups may include teachers of counseling, directors of counseling 
centers, the general public, or makers of public policy (e.g-., legislators). 
Some members of these groups have a tendency to seek simple answers to 
complex problems. They might not be inclined to recognize the limitations 
inherent in conclusions from integrative reviews. 


Pillemer and Light (1980) have suggested that meta-analysis benefits 
the reviewer bys (a) increasing power by increasing sample size, (b) 
obtaining a more’precise estimate of effect magnitudes, (c) describing the 
form of a relationship, and (d) harnessing the results of contradictory 
findings. These advantages improve the generalizability of reviews, but 
must be carefully qualified. If decisions are to be based on such data, 
then estimates of our confidence in the findings should be provided e.g., 
by employing Rosenthal's (1980) “file drawer” calculations for estimating 
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‘presentation of results obviou 


the number of studies with null results not included thatwould be necessary 
to overturn a given conclusion. Meta-analytic findings are temptingly 

clear and simple. We caution that there are rarely neat answers to complex 
questions and that the intricacies of these procedures are difficult to 
convey. Nonetheless, meta-analysis does seem useful in decision making. 


Information about the efficacy of interventions. Meta-analysis has 


been used tovgarner data on the differential effectiveness of one form of 
treatment or intervention as compared to another. This has been done with 
the literature on psychotherapy (Smith and Glass, 1977), sex bias in 
counseling (Smith, 1980), and instructional practices (Kulik, Kulik, & 
Cohen, 1979). To the extent that such treatments are comparable with 
respect to the differential selectivity of subjegts they employ, as well as 
the du.ation, intensity, and frequency of treatment, such comparisons may 
well be useful. However, we agree with Gallo (1978) who suggests that 
decisions about which treatment to employ eventually rest on the relative 
costs to the individual or society of that method. A smal] improvement in 
one arena may be ‘considerably more valuable and important than a large one 
in°some less critical area. The context of these. decisions:may in some 
cases be more important than the findings. 


t * 
Limitations of Meta~Analysis y 


We feel that the principal shortcoming of meta-analysis is the false 
sense of security that use of the approach may engender in the user of the 
aggregated research results. Quantifying study outcomes and applying 
statistical techniques to the resulting effect sizes may imply a greater 
degree of precision than actually exists. In addition, use of the meta~ 


-analysis approach usually results in general summary statements which 
encapsulate the results of a sizable body of research. Cook and Leviton 


(1980) believe meta-analysts run the risk of overlooking “the importance of 
contingency-specifying interactions that in most situations have an infer- 
ential precedence over statements about main effects” (p- 464) and give an 
example. While we agree that meta-analysts may ignore interactions, ye 
feel this can also happen with other methods of research intergration. 
There are a number of other meta-analysis limitations, and these have been 
ably discussed Jackson (1980) as well as Cook and Leviton (1980). 


» For the purpose of using research results for organizational decision 


‘ making, we feel it is the responsibility of the research integrator to 


portray the results of. the oe oe in a realistic manner. This 

y must follow a thorough literature search, 
a thoughtful choice of pertinent study characteristics, reliable coding, 
and careful analysis. Given adequate reporting of research results and 
meticulous implementation of the meta-analysis approach, we believe decision 
makers will have access to more valid information on which ‘to base their 
decisions. 
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